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Meta-analysis does not solve all the problems 
encountered in qualitat^ive research reviews, but it holds out .some 
hope for increasing the reliability and dependability of a reviewer's 
conclusions. Several developments in ^meta-analysis are cause for 
optimism. First, different meta-analysts are doing work in the same 
areas, leading to increased confidence in meta-analytic results. 
Second, meta-analysts are beginning to include their raw data in 
their reports, helping readers to pinpoint the exact studies that 
lead to disagreements in conclusions about an area. Third, reviewers 
are comparing results from unrelated meta-analyses, which will 
eventually lead to a better understanding of the factors influencing 
the outcomes of educational ^ research. And finally, some of the >^Prst 
labuses* that have taken place in meta-analysis have now been left 
behind. (BW) 
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Interest in meta-analysi J has grown enormously since 
Glass first introduced the method in his 1976 AERA 
presidential address. Since that time, approximately 1000 
papers on metaranalysis and quantitative research synthesis 
have appeared in books, journal articles, ERIC dojcuments/ 
and dissertations,^ Approximately one-third of thesi 
reports describe results trom actual meta-analytic studies. 
By almost any standard, the record shows that Glass's 
address has had an enormous impact on educational research. • 

All this should not come as a surprise to anyone who is 
interested in the topic" of research synthesis. Most 
educational researchers know that meta-analysis is a hot 
topic, and they do- not need to hear statistics to get the 
point. Nor is there any mystery about why meta-analysis has 
attracted so much vattention. For a long time before Glass 
introduced meta-analysis, researchers felt uneasy about the 
reliability of qualitative research reviews in education. 
Bu^ no real alternative to this type of review was 
available. 

In an article in the Review of Educational Research a few 
years ago^ Gregg Jackson (1980) listed some of the major 
complaints about qualitative reviews: 

1. Qualitative reviews frequently fail to cite earlier 
reviews on the same or similar topics. 

2. They often report results from only a few^studies in 
areas where many studies are available. v - 

3. They frequently use crude and misleading representations 
of xhe findings of studies. 

4. T^ey sometimes fail to recognize that random sampling 
error contributes to variation in study outcomes. 

'5. They seldom examine relationships between ■ 

characteristics of studi.es and study findings. 

^These estimates are based on two sources: (a) a^ 
comprehensive bibliography ^on meta-analysis compiled by 
William K. Lamb and Dean K. Whitla of Harvard Unive^ity; 
and (b) my recent computer search of Dialog's ERIC 
database for publications that use the term "meta- 
analysis" or "meta-analytic" in their titles or abstracts. 
Lamb and Whitla's comprehensive bibliography lists 261 
publications that appeared before January 1981. My year- 
by-year search of the ERIC system suggests that 
approximately 75% of works related to meta-analysis were 
published after January 1981. It seems reasonable 
therefore to believe that a comprehensive bibliography 
through March 1984 would contain approximately lOOP 
references. ^ 
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6. They usuft^j^fa^ort little about the methods ^used in the 
review |iJp^T^5^g|readers cannot judge the validity of the 
cohclusr 

Meta-analy|^|WH^^ solve all these problems, byt it 
holds out f^^^^fH inqreasing^ the reliability and 

dependability oSI^KW*viewpr ' s conclusions in reviews. It 
promises to ^^'^^^^^s^s more useful.- It is hard for m'e to 
imagine theref ore'^||ii|t there will^be^any turning away from 



meta-analysis in the^ future. It has the look of a survivor. 

/ uses of Meta-Analysis ' 



A number of^^p» in, the meta-analytic 

iiteifature make iw|^.;ftespeci ally optimistic about its future-. 
They are the kinds of things that ^onyince me that the 
approach is growing better and stronger with time. This 
afternoon I would like to mention ttjree of the most " 
promising developments. . 

Emphasis on Replication of Meta-Analvses ' 

First of all> it's become *clear recently that meta-^ 
analyses spawn new meta-analyses in the same areas. ThBt 
may not seem like much, but I find it' very heartening. 
Af ter* Glass introduced the approach in 1976> skeptics began 
warning that meta-analysis might be the death of primary 
Research. " The meta-analyst would analyze' the collected 
results in an area of research. vitality , and interest in 
doing primary research in the area would dry up. The 
average Effect Size could serve as the epitaph for the area. 

Just the opposite, in fact,' has happened. Meta- 
analysis has challenged researchers to pay more attention to 
the areas in which it's been applied. 

1. At least three independent meta-analyses, for example, 
" have been carried out on effects of psychotherapy^, . and 

others are in progress (Landman & Dawes, 1982; Shapiro & 
Shapiro, 1982; Smith', Glassh, Miller, 1980). 

2. At least four independent meta-analyses have been 
published on the effects of coaching for the Scholastic 
Aptitude Test (Der Simonian & Laird, 198.3; J. Ku^lik, 
Bangert, & Kulik, 1984; Messick ^ Jungeblut, 1981; Slack 
& Porter, 1980) . - , 

3. At least three meta-analyses have been carried out on 
effects' of desegregation (Grain & Mahard, 1982; Krol, 
1979; Wortman.Si Bi'yant, in press). 

4. And numerous meta-analyses are available in the area of 
computer-based instruction (Burns & Bozeman, 1981; 
Hartley, 1978; C. Kulik, LKulik, & Bangert-Drowns, 1984; 
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J. Kulik^ Bangert, & Williams, 1983; J. Kulik, Kulik, & 
Cohen, 1980). : : 

Ivsee no evidence that such meta-analytic activities 
have dried up the stream of^ primary research from these ' 
areas. My colleagues and I recently decided to update an. 
earlier meta-analysis -of ours on. coll^ge-^level computer- 
based instruction. We were almost overwhelmed by the number 
of primary studies that have appeared since our first meta- 
analysis wHs done. It is hard. for us to believe therefore 
that meta-analysis has had negative effects on the 
production* of primary research. Jf anything, meta-analytic 
efforts may give primary researchers a better sense of how 
their work fits into a large picture. 

■ • *■ • * 

Publication of Meta-Analytic Data 

A second major development in^the field of meta- 
analysis is also very* encouraging ^o me. In recent years, 
meta-analysta have been growing more open about the data 
that go into their. syntheses. More, meta-analysts today than 
ever before are including in their reports lists of studies,, 
study effects, and study, features. 

I am especially pleased about this development, because 
I > know that the decision to' go public with one's raw data is 
not an easy one for most meta-analysts to make. For one 
thing, it gives others equal ownership of the data the meta- 
analyst sweated over. It also decreases the number of 
publication outlets for manuscripts since m^ny" journals w^ll 
not publish articles with tables of primary studies. It 
also- decreases the .likelihood that other meta-analyses in an 
area will be truly independent. 

The advantages ,' however, more than make up for these 
drawbacks, and my colleagues and' I at' Michigan have decided 
to include our raw data in all our future publications. I 
hope that other meta-analysts will follow us on this c(j)urse. 

The great advantage of going public with one's data is 
to the reader. With meta-analytic data laid out before 
them, readers can see how meta-analysts draw their 
conclusions. ThereK is also a great advantage to meta- 
analysts.' With various meta-analytic data sets laid out 
before them, meta-analysts can usually pinpoint 'the factors 
tha^ cause disagreements^ among reviewers. In the long run, 
this will be of enormous benefit to educational research and 
educational policy. 

Going public with meta-apalyt ic data has already 
cleared up some of the mystery surrounding reviewers' 
conclusions about coaching for the Schola^ic Aptitude Test. 
Each of the four recent meta-analyses on SAT coaching 
reported very similar average effects. ''But the researchers 



conducting the meta-analyses 'drew very ^dif ferent conclusions 
about the potential benefits pf^^ coaching. . With the meta- 
analytic data from all studies available, it is^easy to see 
that the differences in conclusions depended on the weight 
and interpretation that .each meta-analyst gave to a few 
outlier cases. . , 

^ Synthesis of Findings 'from Different Areas 

There is a third important recent , development, in meta- 
"^analysis, and I want to applaud it too. Meta-analysts are 
• now comparing meta-analytic findings f^om different areas to 
see whether certain study fef'atures have consistent effects 
on outcomes. As in so many other things, Glass and his. 
colleagues 4have led the way. in this endeavo^ top (Smith et 
al., 1980).* They have concluded that variation in study 
findings is only modestly predictable f rom stydy 
character.istics. In nearly all instances , less than 25% of 
the variance in study results can be^accounted for by the 
best combination of study, features. According to Glass, 
only a few' study features are consistently related to 
outcomes ,from study to study. 

Since 1978 my colleagues and I have been carrying out 
meta-analyses of findings on instructional technology. We 
recently examined a set of ten of our? meta-analyses for r 
consistency in relationship of study features and outcomes 
(Bangert-Drown^, Kulik, & Kulik, 1984). Our conclusions 
were ^similar to Glass*s..^^ We found that relatively little .of 
the variation in stMy f-indilngs was attributable tq study 
features.* Among the few features that consistently v 
influenced study outcomes were the source of publication and 
the magnitude of the study. Effect sizes were usually^ 
higher in journal articles than they were in: dissertations , 
and they were usually higher in small scale studies. Among 
the study features'^at showed little relationship to size 
of effect was the basic experimental design of the study. ^ 
True experiments and quasi-experimental vevaluations produced 
the same basic results* 

Abuses of Meta-Analvsis * , 

Before discussing areas where progress ystill needs to 
be made, I want to say that meta-analysts are fortunate to 
be able to %race * their work back to Smith, Qlass, & Miller'^ 
(1980) study on effects of psychotherapy. That study stands 
as an intellectual feat of the first magnitude. It "was bold 
in conception, painstaking in execution, and ingenious in 
design. Few studies in the history of educational research 
could equal its inventiveness. 

It &eems to me, however, that Glass and his associates 
in that study were testing the limits of how much could be 
done in analyzing an aggregation of study ' effects . No 



matter what the or iginall- unit of measurement in a study, 
Glass and his colleagues were able to express ,the < , 
measurement in standard deviation units. And. because they 
were able to express all measurements on the same scale, 
they were able to include a. very- il[arge number of comparisons 
in a single statistical analysis. They were thus able to 
use the most sophisticated statistical tools in their, 
analyse?. ♦ „ * ; ' . 

* ^ , * ■ *. 

This bold approach was not without its problems, 
however. One obvious difficulty is' the inflated N in the 
analysis—a sample size much larger than th? number. pf 
independent observation?. When a given study is represented 
two, three, fo\ir, or five times in a data set, it is 
difficult. to determine the amount of ^ error in statistics 
describing the set. It is even more difficult to estimate 
the true correlations among study, features when'sigg^^ 
studies are, multiply represented in a data set.. .The results 
from regression^ analyses on such data sets must u^ually^be 
suspect . ^ 

,But^even more serious is the conceptual confusijon that, 
arises when different types of outcome measures are^lumped 
together. .Gl^ss, Smith, and Miller's overall conclusion 
from their meta-analysis of- psychotherapy was: "jfhe average 
study showed a 0.68 stahdard deviation superiority of the 
treated group over the control groiip." A superiority of 
0..68 standard deviations of what? 'Of palmar sweat? Self- 
' satisfaction? Academic achiev^meift? Job performance? • 
Effects varied- from one. type of criterion to another. 

Few of t^ meta-analysts who 'followed Glass were able 
to match his fngenuity and boldriess, but almost anyone could 
outdo ^him in br:eaking the rules -^on inf lating N's and 
defining criteria precisely; And in the early years, meta-. 
analysts, sometimes did just that. A few mata-analysts went 
to extrem'es to get the biggest possible N's fol: their 
studies. They coded effect siz^s for every possible 
subgroup and subscale in each study. They act6d as though ^ 
entirely different outcome pleasures were completely 
equivalent. And they they gave marginal work the benefit of 
the doii^t,. throwing it into the h^per with work of far 
greater quality. ' • ' > ' ^ . . r 

I think that the \^brst of "these excesses are behind "us. 
'^he probljsm of lumping together different outcomes in one 
analysis seems to ma ta be on its way out. Most meta- 
analyses now look separately at cognitive "and affective* 
outcomes of instruct/ion^ and some carry put a dozen separate 
analyses on a dozen different types of instructional 
butcomes. The problem 'of^dnflatedvN' s seems to me, ho>fever. 
ta be. a more enduring legacy of "the early meta-analytic 
work. Most, meta-lanalysts continue t.o use' as the N in their 
analyses a figure that is a good deal larger than the number 



of independent studies they locate. I hope that some day 
meta-analysts will start giving more attention to the 
results of this practice. • - ' 

Conclusion ^ " 

In conclusion, I want to reiterate that I find several 
developments in meta-analysis a'cause for optimism. ^ First, 
I am encouraged that different meta-analysts are doing work 
in the same areas. This is leading, to increased confidence 
in meta-analytic results. Second, I am very pleased that 
meta-analysts are beginning to include their raw dat^ in 
their reports. That helps reader pinpofnt the exact ^studies 
that lead to disagreements in conclusions about an area. 
Third, I am encouraged that reviewers are comparing results 
from unrelated meta-analyses. I think that this will 
eventually lead to a better understanding of 'the factors 
influencing the outcomes of educational research. And ^ 
finally, I think that. some, of the worst abuses that have 
taken place in meta-analysis are behind us. 
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